Neural networks represent an alternative approach to information processing than the serial algorithmic computers of the type envisioned by John von Neumann. In a computer of the von Neumann type a single processing device executes an explicit program in a step-by-step serial fashion. Many information processing problems can be solved by this approach, as witnessed by the recent explosive rise in the number of computers and in the number of applications susceptible to computer processing.
There are limitations to type of information processing which can be performed by a computer based on the von Neumann model. Before such a computer can be employed to perform an information processing function, a person must understand the nature of the problem to be solved and produce an explicit program of instructions to solve this problem according to an algorithm. An algorithm is a mechanistic or recursive procedure for the solution of a problem. The existence of such an algorithm is a necessary condition for information processing employing a computer based on the von Neumann model. In addition, it is further necessary that a person translate that algorithm into the explicit program required by the particular computer.
There are numerous important information processing tasks which are difficult or impossible to perform using von Neumann type computers. In some cases algorithms have not yet been developed for these information processing tasks. In other cases it may be impossible to develop an algorithm of the type required by von Neumann type computers. Information processing tasks of this type include understanding continuous speech, identifying handwritten characters, and identifying objects having varying orientations.
Problems of this type are of the sort that people can do well while current computers do poorly or not at all. Neural networks are attempts to exploit the supposed organization of the human brain to attack these problems. Neural networks are collections of many individual computing elements called neurons. These neurons are coupled together in complex networks in which any single neuron receives input from many other neurons. The coupling from any neuron to another neuron many be strong or weak.
There are two methods commonly employed in selection of these internal representation of the network. In the most general of neural networks a feedback learning process is employed to change the strength of connection between neurons and to change the transfer function of each neuron to accommodate the particular information processing task at hand. Usually this is done by presenting inputs to the neural network and the corresponding desired outputs, and allowing the network determine the proper internal representation for generating the desired output from the input. An important class of neural networks, including those constructed according to the Hopfield model, do not yet incorporate this dynamic learning process but require preset connection strengths. This permits the construction of large networks by simplifying the construction of individual neurons and their connections. However, this advantage in construction is offset by the requirement that the connection strengths corresponding to a set of desired stored vectors be known.
Typical neuron networks include many individual neurons. Practical uses of neural networks often require hundreds or thousands of neurons. If information processing using neural networks is to achieve wide use, the construction of neural networks must be adapted to permit large number of neurons to be employed. This implies an accommodation between the model of individual neurons and their connectivity and circuits that are readily and abundantly realizable in hardware. Due to the large size of such neural networks, it would be particularly advantageous to construct neural networks in an integrated circuit technology which is relatively inexpensive, such as complementary metal oxide semiconductor very large scale integration (CMOS VLSI).
Hopfield has proposed a neural circuit model in, e.g. "Simple Neural Optimization Networks: an A/D converter, a signal decision circuit, and a linear programming circuit", by J.J. Hopfield and D.W. Tank, IEEE Transactions on Circuits and Systems, vol. CAS-33, No. 5, May 1986, pages 533-541. There are some obstacles in VLSI implementation of this neuron circuit including: 1) the necessary dense connectivity between neurons; 2) the requirement that the linear resistive element from output j to input i (T.sub.ji) be equal to the linear resistive element from output i to input j (T.sub.ij); and 3) the problem of implementing variable linear resistive elements in silicon. The first problem is not different from existing challenges in VLSI designs and can be dealt with using standard and evolving techniques. It is known that the equality of the complementary linear resistive elements is a necessary condition to ensure convergence of all solutions to equilibria only. It is also known that small imperfections generally preserve the convergence to equilibria but may introduce spurious non-equilibria steady states. Thus this requirement is a problem that may be possible to endure. The last obstacle, that of implementing variable linear resistance elements in silicon, appears to be the major obstacle to constructing the existing continuous-time models of artificial neural networks in analog VLSI; in particular, dynamic feedback neural networks of the Hopfield type.